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REWRITTEN MARKED UP COPY 
ABSTRACT 

Disclosed is a A pen type input device with a camera which is has improved usabilityvby 
iiiipioving as a resuh in improved construction of the device. The pen type input device with the 
camera is constiniction adapting adapted for use in detecting both of a horizontally elongated 
object and a vertically elongated object. On the other hand, means for pointing to the object and 
the process content simultaneously, and further means for detecting the fact that the user is 
pointing to the object in an erroneous manner and for teaching a correct manner of teaching of 
pointing to the object depending thereon are provided. 
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PEN TYPE INPUT DEVICE WITH CAMERA 



TECHNICAL FIELD . 
5 The present invention relates to a pen type input device^^ 

with a camera (wit ^jvLmproved usability. More particularly, the 
invention relates to a pen type input device jemployingj^as a 
user interface^ permitting a user to easily call out^ function 
of an information processing apparatus. Namely, the invention 
10 relates to^handy image input device to be used[in th^inf ormation 
processing apparatus . 



BACKGROUND ^RT) OF /*JO£^TC?Aj 

^(^s a prior art relating t^ a pen type input device with 
15 a camera, ^here is)]^a MEMO-PEN ^disclosed in ACM PRESS, HUMAN 
FACTORS IN COMPUTING SYSTEMS, CHI 95 Conference Companion , p256 
- P257 . The MEMO-PEN [isj incorporates a compact camera embedded 
in a penholder to continuously pick-up^ image in the vicinity 
of a penpoint across the penpoint to record |a] handwriting . [Ar?)^^ 
areai{to)be picked-up by the camera of the MEMO-PEN is limited 
to a quite narrow region required f or making^udgment^^direct ion 
of handwriting . 



In the MEMO-PEN, since the camera is embedded in the 
penholder , (^visual ^^^^ camera is^nterf ere^ by^a}hand 

25 of the user^ unless [af^joptical system^as to b^^rxar^^ at a 
position inclined toward the penpoint from a portion^to be gripped 
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by the hand of the user* Therefore, even if ^h^optical system jj 



KJtf- 



having wide visual field is used, there is a limitation £^ofJ f 
distanc^^o plac"^ the optical system|awa^f rom an object^ ( paper ) . 
Therefore, the wide visual field of the camera cannot be 
5 ^^srablishe^. On the other hand, in the condition ^here||the user 
naturally holds the pen, the penholder is usually inclined 
significantly jfroi^up-right position. Therefore, an image to 
be picked up by the camera is a cross shot of the image. 

A function of the MEMO-PEN is to store the handwriting a 

10 of the user written by the MEMO-PEN^to(reproduce]^the handwriting/ 
l^gnde^ the^^^d^ of {s^ome]|inf ormation processing apparatus or to 
recognize J| characters . Namely, during collection of the 
handwriting data , in other words , while the user uses the MEMO-PEN , ^ 
user interface n to call out the function of the information 

15 processing apparatus ^ us ing ^he)j^pen-type input device with(the?^^ 
camera^ would never be executed. Accordingly ,f kind of process 
to be applied for processingjob ject (handwriting) during inputy^ 
is not designated by the pen. Furthermore, in case of the 
MEMO-PEN, since the pen pointer and the position of the 

20 handwritinq constantly match|wit^ each other, it is not necessary 
to ad just ^ positional relationship between/ input object 
(handwriting) and the penpoint. 

As [the prior artj^relating to ^he)/pen-type input device 
with(the)^camera, [there is) a system called(as) PaperLinkj^disclosed 



25 in ACM PRESS, HUMAN FACTORS IN COMPUTING SYSTEMS, CHI 97 



CONFERENCE PROCEEDINGS, P327 - p334 . In the PaperLinl^, [as a) 
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device with which the user inputs a process ob ject^^a pen-type 
input device mounting a compact camera. The compact camera is 
located at a position looking down|^the object (paper surface) 
from right above [iri^the condition[where^the user naturally holds 
5 the pen. The visual field of the camera can be set so that^an 
area of several cm square around the penpoint. The picked-up 
image around the penpoint is input to the information processing 
apparatus to execute various processes depending upon thef 
content , I f t he input ob j ec t is known , |t h^jpredet ermined proces s 

10 depending thereupon is executed. For example,;^ predetermined 

file is opened ^o presentj to the user or execution of {the)y[^ 
predetermined program is initiated. On the other hand, if the 
^put) ob ject|\is not known, it is temporarily storedu|to be used) 
as an argument to |thg|command to be executed later. 

15 A ^ problem to be solved by the present invention is [to} j 

^egrad^f usability (li^jthe device for inputting^an object (using} >\ 
^n imag^ by ^ading)|the compact camera on the pen-type device. ) 

fAnother probleiB- in the MEMO-PEN, since the camera a j 
including the optical system is built in the penholder it is 

2 0 difficult to provide^ wxde visual field for the camera. 

^^ncerning the foregoing PaperLinl^, since[a^axis^^^of^he 
pen and/center axis of the camera are arrange^^on^^ samey^laiS,y^^ 
longitudinally elongated pattern cannot b^|pointed) by natural 

operation.^^ ^j^^^ ^ ^^a^ 

2 5 ^Furthermore input device of the PaperLinljf, since (V 

means for pointingj( the object |is only one kin^, it (cannot be)^ t^^^ 
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possible to designate the process object and kind of process 
simultaneously. 

^ ^.Juq^aji^v^JSM) 

^Also, if theusei^akes designatioHpb ject and the penpoint 
[In the)[construction (wherejithe camera looks down the penpoint 
5 f rom ^he^ above^as in the PaperLmto, the (aesignation)job ject is 
hidden by the penpointyjto make it impossible tc^^akej accurate 

On the other hand, since the penpoint and the^j^signatio^ 
object cannot be overlapped k as set forth above, it becomes 

10 necessary to^[reflect favor] of the user in the method for 
designating the object. Namely, in the pen-type input device 
with fthe)L camera , (a)irelationship between the position of the 
fcfaesignationl object and the position of the penpoint is variable 
depending upon jfavor)/of designation method of the user. For 

15 example, when the user designates a laterally elongated object, 
some users may pointjj/.the center portion, and another user may 
pointflAlower right position. On the other hand, /tilt angle of 
the pen upon pointing^the object may b^^ariable a^ different 



users . 



20 



(disclosure) of the invention 

An object of the present invention is to provide a pen i n 
type input device with a camera which can improve/ usabilitj^. 

In order to accomplish the foregoing obje ct , the present 
25 invention is provided with the f ollowing^jTneansJ. 

In^the present invention, upon installing the camera on 
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the pe^^ [a)/center axis of the camera is arranged/ of f set 
from [a)/center of the^^^^n. When a user points'; a vertically 
elongated object from /right side, the camera is arranged at^^^ 
left side [than al penholder . /^yjthisi/ the penholder will never , ^ . _/\ 
be placed between the camera and the object^^o be nindranc^. 



Conversely, When a user pointsf^a vertically elongated object 
fromj^left side, the camera is arranged atjiright side (than a,U 
penholder. On the. other hand, in place of fixinq the camera 
otts^^^m/j/Stfy towar^left or;right, ^i^onstruction may be^^taken) 

10 to of fset^ toward ;ieft or right jwitlpjipivoti^ ^^^^a^[^^ 
^eft and righ^ within a range from 0° to 90"*^ . 

On the other hand, a compact camera. or a portion of the 
pen^installedj(tlie same] is provided with a process designating 
device for .designating/ kind of process to be fappliedlAfor the 

15 [pointed]j(object- The ^^►rocess designating device may be a 
mechanism for switchingy\color of^penpoint in a multi-color ball 
point pen, for example. By preliminarily setting a 
predetermined process rbyHlitne process designating device, the 
process can be activated simultaneously ^itfi) pointing [ofj^the 

20 process object. The process designating device may be a device 

^ J 

varying^shape or color of the penpoint. Since the penpomt is^i^>^^-*^ 
for pointing J[the object, the user may watch the shape or color 
of the penpoint corresponding to the kind ^^^y/process ^wit^^^^'^-^^ 
capturing the ^ointingj object in a visual field. 
25 On the other hand, A pointing manner teaching means for 

teaching/correct manner of pointing ^r)/the user is provided^ 
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and incorrect pointing detection means for detecting^incorrect 
manner of pointing of the ob ject [of)j(the user is provided. The a 
incorrect pointing detection means may makey^ judgment [of)A 
incorrect manner of pointing of the object by the user when 
a region of the detected object and a region where the penpoint 
appears overlap , for example . On the other hand , when extraction 



of the objectj^f ailed cont/inuously for a predetermined /jtimes 

* . . . 

judgment may be made thatjfmanner of poxntmg the object [ofhthe 

user is not correct. ^./I A * i 

10 Furthermore, in order to [set the favor upo3 j>ointing/the 

object using the pen type input device with; [the) camera ^y the) 
^e^, pointing error correcting means may be provided. The 
pointing error correcting means is different fromj! parallax 
error means j^in a typical tablet display, and is means for 
15 registering [favor) Aof the user concerning / positional 
relationship between the pen type input device with Jtheji^camera 
and the object pointed/by the same. 



BRIEF DESCRIPTION OF THE DRAWINGS 
20 Fig. 1 isifan illustration showing a construction! of an 

overall system employing a video pen ^^-^^^-^^ f/''^^ 

Fig. 2 isj[Jan illustration] showing [a^constructionj||of the 
video pen; 

Fig. 3 i^^n illustrationjshowing the video pen as viewed 
25 from ftheA above; 

Fig. 4 i^|an illustratio^ showing one example of an image 
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picked-up^a ^ine in horizonta]j| [writing^ document by means of 
a video pen; . 



Fig* 5 i^.jain illustrationjshowing one example of an image 
picked-up/\a ^ine in al verticalj( ^iting) document by means of 
5 a video pen; . 

Fig. 6 isj^^n illustration) showing a construction of the 
video pen for lef t-handedy( ag yJ^ewed from (the) above; 

Fig* 7 is^^n illustration forj quantative [discussion for) 
shifting amount of the camera of the video pen; a 
10 Fig. 8 isi[an illustration) showing an example of ^^^shape 

of a penpoint requiring^extremely small shifting amount of the 
camera of the video pen; 

Fig. 9 is^n illustration) showing a construction of the 
video pen; 

15 Fig. 10 isj(an illustratio^ showing a video pen as viewed 

from ^he^ above (setting for horizontal writing); 

Fig. 11 is^^n illustratiojaj showing a video pen as viewed 
from ^e^ above (setting for vertical writing); 

Fig. 12 isj[^n illustratio^ showing one example of an image 
20 picked upjja horizontally elongated object in a setting for 
horizontal writing ; 

Fig. 13 is^n illustrationjshowing one example of an image 
picked up^a vertically elongated object in a setting for vertical 
writing; 

25 Fig. 14 is^n illustratio^showing an overall construction 

of a process designating device; 
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Fig. 15 is|(an illustrationj showing one example of the 
case when an erroneous object is designated by the video pen; 

Fig. 16 is^n illustration) showing an examplql detecting^^ 
overlapping of the penpoint and the object after extraction 
5 of the object; 

Fig. 17 isj^n illustrationj showing one example of a screen^ 
teaching a manner of pointing; 

Fig. 18 i^^n illustrationjshowing one example of a message 
displaying pointing correction means; 
10 Fig. 19 isj[an illustration^ showing one example of a sheet 

used for pointing correction; 

Fig. 2 0 isji^n illustrationjshowing one example of an image 

^ ... 
picked-up^a standard pattern for pointing correction; 

Fig. 21 is [an illustrationj showing [an) / internal 

15 construction of an image processing apparatus; 

Fig. 22 isj[^n illustrationjshowing ^n example of the case) 



@iereja horizontally elongated object as one example of a binary 
image; . 

Fig. 23 i^(^n illustrationjshowing one example of a value 
20 for pointing correction; j ^ a i ^ 5 

Fig, 24 is^[an illustration for explaining ^ pointing 
correction data; . i(\ -A- 4^ 

Fig. 25 isj^ [a^n illustration for explaining] object 



extraction; . 



25 Fig. 26 is^n illustratio^showing one example of an image 

of the extracted object; 
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Fig. 27 is^ (an illustration for explaining aJ tilt 
correction; 

Fig. 28 is|(an illustrationj showing one example of an image 
of^ object after tilt correction; 
5 Fig. 2 9 isj^jan illustrationj showing one example of a 

characteristic amount ; 

Fig. 30 isy^|an illustration) showing one example of a 
structure of a pattern dictionary; 

Fig. 31 is^,[an illustrationj showing one example of a 
10 structure of a process table; 

Fig. 32 is^i^n illustration) showing one example of a 
structure of a operation table; and 

Fig. 33 is^n illustratio^showing one example of a display 
image of the information processing apparatus. 

15 

BEST MODE FOR IMPLEMENTING THE INVENTION 

Fig. 1 shows one example of a hardware construction for 
implementing the present invention. A user may input data to 
an information processing apparatus 102 ^ndl|[ executed a command 
20 of the information processing apparatus 102 employing a pen 
type input device [mount ingjiy a compact camera lOljj. 

Sucl^pen type input device with(th^Jlpamera will be ref erredf 
hereinafter as a video pen 103. When a user points/^ something 
[bj^l^tne video pen 103, a switch mounted at the tip end of the 
25 video pen 103, namely a penpoint switch 104, is turne^^N. The 
information processing apparatus 102 is responsive to/\turning 
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ON of the penpoint switch 104 to take / the image from the camera 
101 of the video pen 103jito execute various processesdepending 
upon the content^. For example, a line in a document is extracted i 
from the image, subject to character recognition^and transfers// 
to a dictionary program. Then, the result is displayed on a 
display 105. ^ 

Fig. 2 shows one example of [a)|structure of the video pen 
103. A penholder 201 [Is aj portion jto gripped by the user. 



A penpoint switch 104 is mounted on the tip end ot the penholder 

10 201. The tip end of the penpoint switch 104 |is)j bar-shaped 
configuration. When the user points [theJ)!object (b^^the video 
pen 103, the bar is pushed into the penholder 201 to turn ON 
[th^j[electrical contact. ^q)|the penholder 2 01, the camera 101 
is mounted so as to pick-upy^image^ around the tip end ^portion] 

15 of the video pen 103. The camera is mounted on the penholder 
103 by a camera holding member 202. The camera 101 is mounted 
at a position^to vertically look downj(a portion around the tip 
end as much as possible when the user holds the video pen in 
a fanner/ holding ^he]i|pen. As the camera 101, a commercially 

20 available compact video camera may be used. The compact ^ide)^^^^^^^ 
camera employing a 1/4 inchitjiK image pick-up element has a circular 
cross-section less than or equal to 1 cm in diametei^jj a length 
of about several cm. On the other hand, the penholder 201 ^s^^'^ ^ 
circular bar-shaped cross-section about 1 cm in diameter 

25 similar^^ toj^typical pen. , 

Fig. 3 shows a jcase^looking down from the right above 
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with Rising] the video pen 103jfin the same condition as that^ 
the user holds the same. Inj(the)case, since the camera 101 is 
arranged in)i^ vertical direction, it can be seen in circular form 
as in ^he) cross-section. Here, the point to notice is that the 
5 penholder 201 and the c^era 101 ar^^rra^^ed on the sam^^lai^. 
The camera 101 is arrangedjof f set^jfi^^upper left direction from 
the^fclain ofl| which the penholder 201 falls. ^og!j|pointing^^he) ^ pj 

[l.ine of thej verticaly^ document jbyjjtne video pen 103, this [Is}/ 
^r consideration fo^ prevent 5J6^ the objective line from being 

10 hidden by the penholder 201. t) ^\ 

Fig- 4 shows an example of the image of (the picked upjA 
line inj^[the horizontal writing) document^ by the video ^in)y(l 03 
shown in Fig. 2. A penpoint 401 of the video pen 103 appears 
at a position slightly offset [to]jtlower right direction from 

15 the center of the image. On the other hand, at a center portion, 
objective pattern 4 02 pointed/ by the user is picked up. 

[On the other handj) Fig. 5 shows an example of the image 
of ^he))^line in ^Jie vertical writing^document picked up by the 
video pen 103 shown in Fig. 2, similarly. The penpoint 401 of 

20 the video pen 103 appears at the same position as that in Fig. 
4. This is natural since the positional relationship of the 
camera 101 and the penholder 201 is fixed. In Fig. 5, the 
objective pattern 501 appears to extend vertically at the center 
of the image. At this time, since the camera 101 is arranged^ 



25 offsetWflKB® toward;|left from the penholder 201, the penholder 
201 never (Xnt erf eresjjjthe camera j|to hide the objective pattern 
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501 in the vertical^ writing document. 

Fig . 6 shows a method for installing the camera upon forming 
the video pen 103 for left handed^ similar!^ to Fig. 3. In^the 
video pen 1 03lshown in Fig . 3\is[premised injright handed operation 
5 and is assumed to point^^h^ vertically elongated ob ject^by;right 

hand f romjright s ide . In order to pointfthe vertically elongated . , 
ob jectjbyjlef t handf rom^eft side, the camera 101 maybe installed/^ 
offsetl^ifl^ toward yright from the penholder 201 .^^^y) thi^, even 
when the vertically elongated object is pointed^from the left 

10 side by the video pen 103, the object will never be hidden by 
the penholder 2 01. ^Ji^ 

Fig. 7 shows how much offset from the center* fplain) of a 
the penholder 201 is required for the camera 101. ^o as)/^not 
{ho^shide the vertically elongated o^j^ct |by the penholder 20^, 

15 the camera 101 has to be offset g.n)>a magnitude greater than 
or equal to half of the thickness 701 of the penholder 201 at 
the portion picked up by the. camera. However certain shapes 
of the penpoint 401, ^he) ji^smaller offset magnitude/ (can) be 
sufficient. For example, when i^the penpoint 401^^ in a range 

20 (appearing in)|;the image of the camera ^^j^the shape as illustrated 
in Fig. 8, the of f se^magnitude can be greater than or equal 
to 0. Accordingly, Jloff set magnitude of the camera may be 
adjusted so that the offset magnitude is greater than the width 
of the portion of the penholder 201^in a range ^appearing 

25 the image of the camera^inc lined toward the object to be pointed 
[tha^J(^the center line of the penholder 201. 
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Fig. 9 shows another example of the structure of the video 
pen 103* (a portioftfiof the pen holder 201 is the same as that 
shown in Fig. 2^ but is^Q^ifferentiatedJ in/mounting [methodj of 
the camera 101. 

Fig. 10 shows a (casej^looking down from the right jabovej 
with[raising|the video pen 103^shown inFig. 9 in the same condition 
as thatj^the user holds the same. In this case,j||dif f erent from] 
the case of Fig. 3, the camera 101 is arranged on^^th^e samej^lain^p^^^"^ 
with the penholder 201. Accordingly, there gTOul^ibe ^ised) no 
problem in pointing/the object inihorizontal/fwritingl. However, 
upon pointing^^he) vertically elongated object, sucli as^line 
of (the vertical writing^jdocument , the pen-holder interferes^. 

For solving this, in the video pen 103 shown in Fig. 9, 
camera 101 (is'^hifted toward/left withj||rotating ove^ 90° with 
15 respect to the penholder 201 ..Fig. 11 shows a condition^locking 
down the camera 101/of f setiiJ5fft!ftj( toward^^left with rotationjj. The 
camera holding member 202 is bent at (thSiintermediate position, 
and the camera is rotated over/ 90° to swing out towardjf lef t . 
In this condition, upon pointing^he)j[/ertically elongated object, 
20 the penholder 201 will never hide the object. 

Fig. 12 shows one exampLe of the image /picked up the]c^ ^ 
horizontally elongated object^n^thecondition^^jf'ig . 10 . There 
is no significant difference {as^j^the case of Fig. 4 except for 
the position ^ppearing)/the penpoint 401 being slightly ^we:^ (r^^^^^^ 
25 ^de of] the center of the image. Namely,, upon pointing/ the , aA 
horizontally elongated object, it can be said that^the video 
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pen 103 shown in Fig. 2 and the video pen 103 of Fig. 9[ha^ 
^ttle difference uponj^pointingj) \the) horizontally elongated 
object. 

On the other hand. Fig. 13 shows one example of the image 
5 fpicked up the^vertically elongated ob jecty^jlnj the condition[of)-<2A<^A^- 
Fig. 11. In this case, since the camera 101 is rotated overifrt* 
90° , the penpoint 4 01 appears atj^wer)left side of the center 
of the image . On the other hand, the vertically elongated object 
appears as elongated inj^horizontal direction of the image. The 

10 image picked-up by ^he)jcommercially available video camera is 
typically elongated in horizontal direction. In the 
construction of the video pen 103 shown in Fig. sQ^even upon 
pointing ^ejjvertically elongated object, the image of the image 
pick-up element can be used effectively. 

15 Namely, upon pointing (the)j[verticalLy elongated object 

^y^j^the video pen 103 shown in Fig. 2, onlyjobject falls within/^ 



shorter sides of the image^as shown in Fig. 4. However, increase 
of the video pen 103 shown in Fig. 9, it becomes possible to 
input the object j^f alls within the longer sides of the image^ 



2 0 as shown in Fig. 13 



The video pen 103 shown in Fig. 9 encountersji in that, 
upon pointing {the)^vertically e^ngated object, the user has 

_ to change the arrangement of |h^|icamera 101. Accordingly^ it is 
desirable to employ the construction shown in Fig. 2 when 

25 resolution and image pick-up range of the camera is sufficiently 
large, ^to employ the construction shown in Fig. 9 otherwise. 
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In the construction of the video pen 103 shown in Fig. 

<^ 

9f so that the information processing apparatus 102 may make/i 
judgment j( whether the object is horizontally elongated or 
vertically elongated, it becomes necessary to read where the 
5 camera 101 of the video pen 103 is set. This may be done by 
electrically [rea^jthe condition of the camera holding member 
202 or by making; judgment from the picked-up image. Namely, 
by checking where the penpoint 401 appears in the image, the 
position of the camera 101 can be identified. If the penpoint 

10 402 appears atf central lower side of the image* as shown in Fig. 
12, (It is the)jcase where ^h^jhorizontally elongated object is^ 
picked up. On the other hand, when the penpoint 401 appears 
at/lower left side of the image^as shown in Fig. 13^ ^t is^ the)^ 
case where the vertically elongated object is pointed^f rom^right 

15 side. Also, when the penpoint 401 appears at^lower right side 
of the image, ^ is theW:ase where the vertically elongated 
object is j^f^ointed;f rom/lef t side. 

In the portion of the penholder 201 of the video pen 103, 
a process ^^^^^^^^^^^ device^J^ O^l^^fo^ des^i^ating^j kind of [the) 

20 process to be^^pplied) f or the^ointedjpb ject is provided. The 
process designating device 1401 is[sucfl amechanism for switching 
pen color in a multi-color ball pointed pen, for example. The 
user selects a color corresponding to the process to be activated 
by rotating a rotary portion 14 02 at the upper end of the penholder 

25 201. 

By preliminarily setting (the]i^predetermined process ^yj^/^^^^^ 
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the process designating device 1401, the user may activate the 
process simultaneously with designation of the object to be 
processed. The process designation device 1401 may also vary^ 
shape or color of the penpoint 401. Since the penpoint 401 is 
5 the portionj^to point/the object, the user may watch the shape 
or color of the penpoint 401 corresponding to the kind of process 
^ithjijjwatching th^jobject |to poirvp. 

For reading^condition of the process designating device , 
namely^ which one off pens is selected, a construction employing 

10 electrical contacts may be considered, for example. In such^ 
case, upon turning ON of the penpoint switch 401, the condition 
of the processing device 1401 may be read out. 

On the other hand, since the penpoint 401 falls within 
the visual field of the camera 101 , it may be possible to identify 

15 the kind of ^he) process by image processing without usingA 
particular electric signal upon processing the image from the 
camera 101. Since the positional relationship between the 
penpoint 401 and the camera is known, the position of the penpoint 
4 01, particularly the pen in the input image from the camera 

20 lOljcan be calculated preliminarily. Durinq^process ofjthe image 
processing , by checking the color present at the current position 
kind of the currently selected process can be easily 
discriminated . 

In a user interface method according to the present 

25 invention, the user inputSj^the^rocessjob ject to the information 
processing apparatus 102 ^y an imagej using the video pen 103. 
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Accordingly/ in order to accurately input^the [process] object , 
it becomes necessary to acquire the correct object designation 
method ^yjfjthe video pen 103. Namely , the penpoint 401 of the 

— video pen 103 should not hide the object^ and the penpoint 401 
5 and the object should not be too distant. 

The userjusing the video pen 103|ai^^the first time, tends 
to overlap the penpoint 401 [overj^the object^as shown in Fig. 
15. This condition will not create|f a problem when the object 
to be pointedj^is/ displayed imagejll of the system, such as a pen 

10 input ComputerLand has means 2^o:^jtinput coordinates on the displayy^ 




pointedjiby the pen. However, in the user interface method. as 
\ln^he present invention, when thej^ointe^ object is extracted 
from the image by picking up ^hej^ortion around the penpoint 
401, i^^^^hxDul problemjto be hidden the pointin 

15 objecty^^by the penpoint 4 01. 

As set forth above, [fpr a measurj/ in the case where/user 
places the penpoint 4 01j(overlap|^fifiiS(ia9 with the object, it becomes 
necessary^means for teachingyjmanner of pointing by detecting/ 
occurrence of overlapping^jplacementj of the penpoint (pver][/\ the 

20 object and teaching the correct manner of pointing. a 

Overlapping of the penpoint 40l[and the pointing^ob ject 

can be detected in the following manner. Namely, a region 

extracted asf object and a region where the penpoint 401 should 

appear are compared toy^^ake judgment thatj the user^placed the 

25 penpoint; overlap^fiSSjiK^ (wit %the object* if both have/ overlapping 

A 

portion. Fig. 16 shows one example where a region 1601 extracted 
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asi^object and the projection 1602 of the penpoint 401 overlap* 
The region where the penpoint ^O^^can be derived preliminarily^ 
since the positional relationship between the penholder 201 
and the camera is known. Upon detection that the user overlaps 
5 the penpoint 401^, a teaching display [of teaching^jjthe user the 
correct manner of pointing^the object is displayed. JLt is also 
poss ible to^ispla^the teaching display[o:^he manner of pointing 
when the failure of extraction of the object |ls faile^jin series 
(for the)/predeterminedji times , for example, three times. 
1 0 Fig .17 shows one example of a teaching image to be displayed 

on a display 105 of the information processing apparatus 102. 



The pointing ^jnn teach^g image teaches jtojthe user to point 

several mmj^lower side in/case o^^^lip^^^orizontall elongated 

object and several mm horizontally^right side in; case of ^he}^ 

15 vertically elongated object. 

On the other hand, in the pointing manner teaching image, 

a pointing correction button 1701 is provided for calling the 

pointing correcting means^which will be discussed^^ater . 

When the user points^the object [by means o^/the video 

20 pen 103,llmanner of pointing is [variable iry^each individual. 

For example, upon pointing j^hej^horizontally elongated object, 

some useismay pointfthe lower side of the center of the object, 

and another user may pointj[lower right side of the object . Also, 

the tilt angle of the pen relative to the object may be [variable 

25 [xi^A^ach user. For example, upon pointing jthej^ horiz^tally 

elongated object, some users may point/ the object fromj^right 

A 
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below, and another user may pointji the object from the lower 

right. Furthermore, ^distance from the object to be pointed^and 

the penpoint 401 may be^^l^ariable per) user. For example, upon 

pointing ^^|horizontally elongated object, some user may point 

5 immediately^lower side of the object and another user may point 

^he} (position about 1 cm below the object. 

As set forth above, /different (jpfj manner of pointing {of") jfe- 

<^ 

the ob ject^depending on the user^has to be ref lected[to)|parameter 

upon extraction of the object jpointed by the use^. Also, in 

10 some cas^ it may become necessary (f o^/v, ad just ftift^i positional 
A 

relationship between the penholder 201 of the video pen 103 
and the camera 101. /^e^, in the user interface method according 
to the present invention, there is provided means for 
preliminarily registering^ difference [o:^y(pointing by the user 
15 ( hereby referred to as "pointing" ) . This means will be referred 
to as pointing correction means. 

Upon activating the user interface method according to 



20 



the present invent ion. at first, the pointing correction means 

H 

is called upon/J depression of the pointing correction button 
1701^ set forth abovet^and so fort^. The user may register his 
{favo:^j^by ^eansjpf the pointing correction means in the following 
^^matte^, for example. When the user activates the pointing 
correction means , a message (showingjj(in Fig. 18 is displayed 
on the display 105. According to the message, the user points/^ 
25 the object onji pointing correction sheet shown in Fig. 19 fhyj 
I^ieans oBithe video pen 103. The pointing correction "sheet is^ 
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simpleji paper [printed the^horizontally elongated objec^. The 
object, shape and the size of the standard pattern 1901 are 
preliminarily registered in the information processing 
apparatus 102, 

5 When the user pointsjthe standard pattern 1901 [by means]ji 

^f) the video pen 103, the image shown in Fig. 20 is obtained^ 
as the input image. The information processing apparatus 102a 
read out inf ormationj^what portion of the ob ject(,)the use:^ pointed ^ 
the penpoint 4 01, andjwhat angleQthe penholder 2 01 is tilted 

10 ^po*R)j pointing by performing image processing for the image. 
The (favorjj^of the user relating to designation of the object 
thus read out is used as a reference value upon extraction of 
the object to be processed from the input image by the information 
processing apparatus 102. 

15 In t he ^(shown) embedment , [^>(d is t a nee between the object 

and the position pointed/by the user (object minimum distance) 
anc^tilting ^standard tilting ;^of the video pen 103 are registered 
as/vuser • s ^^avo^. ^Wanner of deriving [those)}wiJ.l be discussed 
in detail later together with/discussion ^o^^ob ject extracting 

20 portion of the information processing apparatus 102. 

Fig. 21 shows one example of the overall construction 
for implementing the present invention|iJ and illustrates^' 
particular internal construction of the information processing 
apparatus 102. Hereinafter, / operation of i respective 

25 components will be discussed. 
(1) Video Pen Interface 2101 
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When the user pointsj^the object to)^ input by the video 
pen 103, the penpoint switch 104 mounted at the tip end of the . 
video pen 103 is turned ON. The video pen interface 2101@etects^ ' 
turning ON of the penpoint switch 104;^{tq) take^one frame of/image 
5 from the camera 101 ^o^transf er^^ to a binarizing portion 2102. 
The image transferred to the binarizing portion 2102 is a color . 
image having 320 dots Lx^ horizontal direction and 240 dots inA 
vertical directionjand^i^l^^color (number^of 24 bits per one pixel. 

On the other hand, the video pen interface 2101 reads 
10 in the condition of the process designating device 14 01 attached 
to the video pen 103|ito write^^in the process designating buffer 
2103. Data written in the process designating buffer 2103 Ls^^tl^ 
number of the pen selected by the process designating device 
1401, for example. However, when^riumber is 0, it represents 
15 the condition that no process is designated. 
(2) Binarizing Portion 2102 

The binarizing portion 2102 converts the input image into/ 
binary image^and transfers the resultant binary image/ to an 
object extracting portion 2104. Fig. 22 shows one example ot^\j^ 
the binarized image y\ transferred to^ the object extracting 
portion 2104. 



It should be noted thatj in the construction where the 
condition of the process designating device 1401 cannot be read 
out from the electrical contact, in advance of binarization 
25 of the input image, /.color of the region where the pen appears 
is checked to determined what color ^fj pen is selectedj^to write 
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the number of pen selected in the process designation buffer 
2103. 

(3) Object Extracting Portion 2104 

The object extracting portion 2104 is a j^ortio^^ for 
extracting the jprocessj object]^ from the binarized image/ 
transferred from the binarized portion 2102 . In the transferred 
binarized imag^, various matters other than/ object appear^. 



For example, /penpoint 401^^appear^ and matters written near the 
object may appear[V]. fujr^tion of the object extracting 

10 portion 2104^extract</ onlyjlob ject^rom the transferred binarized 
imag^. The image^ isj| transferred to a tilt correcting portion 
2105. 

[Pavo^^when the user points/the object, is stored in the 
pointing correction buffer 2112. The content of the pointing 

15 correction buffer 2112 is a set of values/ shown in Fig. 23, 
namely^[a setj of/object minimum distance 2301 and a standard 
tilt 2302. The object minimum distance 2301 is derived with 
reference to how much distance is provided between the penpoint 
401 and the object when the user points/the standard pattern 

20 1901. The object minimum distance 2301 (can bg) can be set as 
a length (number of dots) of/ line segment, D ^in Fig. 24, for 



example. Also, the reference tilt 2302 is a value expressing 
how much the user tilts the video pen 103[Vj and can^^etermine/ 
the magnitude of the tilt A in Fig. 24, for example. An initial 
25 value of the object minimum distance 2301 is 20, for example, 
and the initial value of the standard tilt 23 02 is 0 , for example . 
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In case of Fig. 24, the object minimum distance 2301 is 18 dots 
and the standard tilt 2302 is 32° . 

A processQ in which the object extracting portion 2104 
extracts the object from the binary image will be discussed 
with reference to Fig. 25 • It should be noted that, in the 
following discussion, the coordinate system ^^|the image has yXf 



.nq c 
at|u] 

used in the following discussion are CAMX representing/ X 



origin atjupper left position. On the other hand, the constants 



coordinate of the penpoint 401 the image, CAMY representing 

10 Y coordinate^and D representing^ob ject minimum distance 2301. 
Since the positional relationship between the penholder 201 
and the camera 101 is known, CAMX and CAMY are values derived 
preliminarily. D is a value which can be read out from the 
pointing correction buffer 2112^ as set forth above. 

15 At first, the object extracting portion 2104 searches 

the image of the object f rom ^he) a point S (CAMX, CPMY-D) uJLJl^^ 
of f setiliiSiJ)^ f ronycoordinate (CAMX, 

CAMY) of the penpoint 401. Namely , ^ along the line segment L 
in Fig. 25|^ the image of the object is searche^. At this stage, 

20 if any pixel considered as belonging in the object cannot be 
found, extraction of the object fails. Here, (foundj^pixel of 
the object is stored as the object region. 

Next, the object extracting portion 2104 checks around 
the object region and takes^.the pixel within a predetermined 

25 distance from the object region in^new object region. The 

predetermined distance is 10 dots, for example. Associatf 

A 
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with this process, the object region is gradually expanded 



radi 



The object extracting portion 2104 terminates^expansion process 
at a |timing)where no further pixel to be taken is present. At 
^timing where^the expansion process is terminated,/ ag^xegate 
5 of the pixels stored as the object region is judged asyobject 
pointed j^by the user. . ^ 

At a Riming where^j^the object extracting portion 2104 
completes extraction of the object, the extracted object region 
and the region where the penpoint 401 appears are compared. 
10 When both have^ overlapping portion, judgment can be made that 



the user^placed the penpoint 4 01/overlappi^as§ with the object. 

For example, J [i^ is the case shown in Fig. 16. Accordingly, in 

suchj^case,^ pointing manner teaching image is presented to the 

user to teach the correct object pointing method. Also, if 
15 extraction of the object [ls)j|f ailed continuously, the pointing 

manner teaching image is presented to the user to teach the 

correct object pointing method. 

For example, when the foregoing method is applied to the 

binarized image shown in Fig. 22, a pattern shown in Fig. 2 6 
20 is extracted a^object. The image [as result ofjj^extractiony is 

transferred to/ tilt correcting portion 2105. 

(4) Tilt Correcting Portion 2105 . 

The tilt correcting portion 2105 is^for calculation of^^^ 

tilt of the object from the image of the object transferred 
2 5 from the object extracting portion 2104 and/correction thereof. 

At first, the tilt correcting portion 2105 calculatesK^i^'/ 
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tilt of the main axis of the received object. An angle R in 
Fig, 27 fis ajj^tilt of the main axis. In the case of the [shown)/ 
example, the tilt is -28° . This is the tilt of the object in 
the image picked up by the camera 101. 



Next, the tilt correcting portion 2105 calculatesQjttilt 
of the object relative to the paperQ on which the object is 
written on the basis of the tilt of the main axis^ calculated 
as set forth above^and the value of the standard tilt in the 
pointing correcting buffer 2112. This tilt will be referred 



10 to as^actual tilt in the following disclosure. Particularly,^ 



.A 

A 

sum of the tilt of the object in the image and the value of 
the standard tilt becomesjtactual tilt . In the ^ownj embodiment , 
since the tilt of the object in the image is -28° and the standard 
tilt is 32 , the actual tilt derived therefrom is 4° . 

15 When the actual tilt is in a range close to 0, e.g. from 

a. 

-45° to 45° , the tilt correcting portion 2105 makes/, judgment 
that the object is horizontally elongated^to rotate the image 
of the object so that the tilt of the main axis [pfjf^ the image 
becomes 0. Accordingly, in case of the example set forth above, 
20 the image is rotated over -28° . The image of the object after 

rotation is shown in Fig. 28, Conversely, when the actual ^A^l^^^ 

is out of the range set forth above, the tilt correcting portion 

^ J 
2105 makes/ judgment that the object is vertically elongated, •^''^ 

A 

the image of the object is rotated so that the tilt of the main 
25 axis of the image becomes 90° , 

The tilt correcting portion 2105 transfers the image of 



the object corrected(the txl^as completed^the foregoing process , 
to a characteristics extracting portion 2106, On the other hand , 
the image of the object correctec^^the^tilt is stored in an object 
holding portion. The object holding portion holds the image, 
and if j already held image is present, {it)j|is abandoned. 
(5) Characteristics Extracting Portion 2106 . 

The characteristics extracting portion 2106 is ^ for 
extracting the characteristic amount from the image of theobject 
fed from the tilt correcting portion 2105. The characteristic 
amount may be|pixel number in^uded in the object,/ size of^ 
externally inserted rectangle, /gravity center position and so 
forth. These characteristic amounts are used for identifying 
the object later. One example of the characteristic amount is 
shown in Fig. 29. An object pixel number PIX-NUM, an object 
external insertion width BOX-WIDTH, an object external 
insertion height BOX-HEIGHT, a gravity center X coordinate COG-X, 

iii 

a gravity center Y coordinate COG-Y and so forth formji'^ 
characteristic amount. These values are all integers* 

The characteristic amount extracted by the 
characteristics extracting portion 2106 is transferred to an 
object recognizing portion 2107. 



(6) Object Recognizing Portion 2107 . a 

o 

Theobject recognizing portion 2107 is ^(portioi^f or making^ 
judgment/. whether the object ^njcurrentlyy^process is registered 
or not, using the characteristic amount transferred from the 
characteristics extracting portion 2106. 



Ok 
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The object recognizing portion 2107 compares a plurality 
of the characteristic amounts registered in a pattern dictionary 
2113 and a characteristic amount transferred from the 
characteristics extracting portion 2106 to check whetheiji close 
5 characteristic amount is present or not . The pattern dictionary 
2113 jis) consis-^|B* of a dictionary item number region holding 
the item number included , anc^ pattern dictionary item more than 
or equal to 0 as shown in Fig. 30. Furthermore, the pattern 
dictionary item ^sj cons is t^54l of a characteristic amount region 
10 holding the characteristic amount and an object identification 
number region. In the [^hown] ^embodiment , as the object 
identification number stored in the object identification 

Ok 

number region, ^natural number (1, 2, 3, .•-) is used. 

The object recognizing portion 2107 retrieves the pattern 

15 dictionary item having ^he)j|characteristic amount close to the 
input characteristic amount^ and transfers the object 
identification number stored in the object identifying number 
region of the corresponding item to the operation executing 
portion 2108, When /.registered pattern having (th^^*^ 

20 characteristic amount close to the input characteristic amount 
is not found, the object recognizing portion 2107 transfers 
-1 as the object identification number to the operation executing 
portion 2108. 

(7) Operation Executing Portion 
25 The operation executing portion 2108 is a portion) t>^-''^ |^^ 

executing ^h€)^predetermined process on the basis of the object 



- 28 - 

identifying number transferred from the object recognizing 
portion 2107. The process content will be discussed hereinafter . 

When the object identification number is -1, namely when 
the input object is not [the^yialready registered pattern, the 
5 operation executing portion 2108 reads out the object image 
stored in the object holding portion jto store/in the pattern 
buffer 2109. At this time, when^uaage already stored in the 
pattern buffer 2109 is present , /newly stored image is added 
to the right side . The object pattern accumulated in the pattern 
10 buffer 2109 is held until/used in ^h^/later process. 

On the other hand, when the object identification number 
_ is -1 , namely^ when the input object is not ^h^already registered 
pattern, the operation executing portion 2108 reads out the 
pen number held in the process buffer. When the pen number is 
15 0, nothing is done^and )[the pen number is not 0, the process 
corresponding to the number is executed. The pen number and 
operation arej^(corresponded by the] process table. 



The process table [Is) cons ist^i® of a process item number 
region holding the item number included therein, and greater 
20 than or equal to 0 of the process designation items, as shown 
in Fig. 31. The process designation item ^sj cons is t^^ifl of the 
state region holding the pen number and the process designation 
region holding the operation to be executed when the number 
is set. The operation executing portion 2108 fls) checks the 



25 process corresponding to the number using the process table^ 
when the pen number stored in the process designation buffer 
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is not 0 . 

On the other hand, when the object identification number 
transferred from the object recognizinq portion 2107 is not 
-1, namely^the input pattern is ^h^patternj[already registered, 
5 the operation executing portion 2108 checks which operation 
^s^ correspond^dSt to the object identification number to execute 
the operation. 

r. -A 5 

The operation table [is) cons isl^ttia of^foperation item number 
region holdingjitem number included therein and more than or 

10 equal to 0 of operation designation item. The operation 

5 

designation itemG^slconsisttfiS of the object number region holding 



the object identification number and the operation region 
holding the operation to be executed upon detection of the object . 

The operation executing portion 2108 checks whether^ 

15 operation designation item having^jthd) number j the same as the 
object identification number given from the object recognizing 
portion 2107 is present in the operation table* If present, 
the operation region of the corresponding operation designation 
item is read out and executed. 

20 I Operation which the operation executing portion 2108 may 

execute^ is^OPEN operation, for example. When the operation ^ 
to be executed by the operation executing portion 2108 is/OPEW|^, 
the operation executing portion 2108 at first stores the pattern 
stored in the pattern buffer 2109 in a pattern image file of 

25 a given name (e.g. "patterns.bmp) • Next, referringfja file name 

or a program name stored^ following ^o] the operation name OPEN,^W 
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file is opened or/ program is triggered. 

The program activated by the operation executing portion 
2 1 0 8 (mayTir ead the pattern image file and is used [Injij^arbitrary 
process* For example, ji image stored in the pattern image file 
5 is regarded a^character string to attempt character recognition • 
Also, the program to be triggered by the operation executing 
portion may usej[arbitrary function of the information processing 
apparatus. For example, it may be possible to call a function 
encyclopedia program or a function of a map program loaded in 
10 the information processing apparatus 102. 

By combining the foregoing, as a program to be triggered 
from the operation executing portion 2108, a program to perform 
character recognition of the content of the pattern buffer 2109/ 
to transfer/ to the encyclopedia program to display a checked 
15 meaning, can be realized. 

Fig. 33 shows one example of a screen image displayed 
on the display (^1 05 of the information processing apparatus for 
implementing the present invention. The image^ which the user 
inputs by means of the video pen 103, is displayed in a process 



20 step displaying region. In the process step display region 



Step of processing the input image may be displayed. The right 
side of the process step display regior^ is a system message 
region. The message from the information processing apparatus 
to the user is displayed in this region. On the lower side of 
25 the screen, there is a pattern buffer region. In the pattern 
buffer region, the process object which the user pointedys^[by^ 
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[means of] the video pen 103|7| and the information^ processing 
apparatus 102 stores temporarily^ is]; stored. 

With the present invention, j| object can be input b^natural 



operation. On the other hand ,j number of operations of the user 



5 can be reduced significantly. Furthermore, it [can|(( prevent 
repeated erroneous object pointing {method/ by the user to cause^ 
disturbanc^. 



